Piloting and Evaluating a Redesigned Self-Directed Learning Curriculum

Satoko Watkins, Kanda University of International Studies, Japan
Neil Curry, Kanda University of International Studies, Japan
Jo Mynard, Kanda University of International Studies, Japan

(Introduction by Katherine Thornton, column editor) 

Watkins, S., Curry, N., & Mynard, J. (2014). Piloting and evaluating a redesigned self-directed learning curriculum. Studies in Self-Access Learning Journal, 5(1), 58-78.

Download paginated PDF version

Introduction to the Column

Katherine Thornton, Otemon Gakuin University, Osaka, Japan

In the fourth installment of the column following the self-directed learning curriculum development project at Kanda University of International Studies, Japan, Satoko Watkins, Neil Curry and Jo Mynard detail the process of conducting a pilot of a possible self-directed learning curriculum for freshmen students, that would meet the needs and principles established in the previous two installments. This pilot represents quite a shift for the learning advisors (LAs), as it would bring what has up until now been a largely self-study course into the mainstream classroom environment, taught by LAs. The installment offers an insight into the strengths and potential weaknesses of such a course, and how students responded to it.

Piloting and Evaluating a Redesigned Self-Directed Learning Curriculum

Satoko Watkins, Kanda University of International Studies, Japan
Neil Curry, Kanda University of International Studies, Japan
Jo Mynard, Kanda University of International Studies, Japan

In the previous three column installments, the Self-Access Learning Centre (SALC) curriculum development project, delivered by the Learning Advisor (LA) team at Kanda University of International Studies (KUIS), was introduced. The installments covered: framework and environment analysis (Thornton, 2013), needs analysis (Takahashi et al., 2013), and principles and evaluation of the existing curriculum (Lammons, 2013). The curriculum development project was undertaken systematically, based on an adaptation of a curriculum design model originally intended for language curriculum design (Nation & Macalister, 2010). The present and fourth installment documents Phase 3 of the process illustrated in Figure 1: Design & Piloting. This phase included the following:

• Re-designing the delivery format, content, sequencing, and assessment ready to pilot
• Piloting the new curriculum with one freshman class
• Evaluating the pilot
• Analyzing the results
• Conclusions

Figure 1. The Curriculum Modification Framework (adapted from Nation and Macalister, 2010 by Thornton, 2013)

Re-designing the Format, Content and Sequencing, and Assessment

Format

Format refers to how the curriculum would actually be delivered to the students i.e. as an optional outside class self-study course, as classroom-based content etc. and was the first item that needed to be decided. It was decided that the curriculum should be piloted with one Freshman English class over the course of the first semester, which involved firstly compulsory in-class input sessions taught by LAs, and secondly an optional outside class self-study supported by LAs. There could have been other delivery formats; piloting a new optional learner training course which would replace the existing First Steps Module (FSM), asking teachers to pilot the material in class without the assistance of LAs, or offering a series of optional workshops to instruct students on self-access learning, but classroom delivery by LAs was chosen for three main reasons. Firstly, offering SALC curriculum content during class time would be a more practical way to reach all freshman English students in the future, as opposed to offering only an optional course. Secondly, this format enabled the LA team to adhere to the established principles of format and presentation for a self-directed language learning course (see appendix A). For instance, the principles suggest ensuring that the course caters for different learning styles and students’ preferences, the input and experimental learning are balanced, and the students are provided with opportunities to interact with peers. These principles were established based upon the results of the environment analysis (see Thornton, 2013), the needs analysis (see Takahashi et al., 2013), focus groups (Hasegawa & Thornton, forthcoming), consulting literature, and established learning outcomes in the previous stage of the curriculum design project (see Lammons, 2013). In particular, the format allowed for numerous opportunities for students to work with peers and for different kinds of reflection and interaction (written, face to face, peer, etc.). Finally, as this was part of a research project, the LAs could work closely with the students and the teacher in order to make observations and gather information that would be important for later decision making.

Thus, the pilot course outline was developed as identified in Table 1 and consisted of two phases. First, there was a compulsory phase where the Core Course Content (CCC) would provide the self-directed learning (SDL) training that was identified as being crucial during the needs analysis phase of the project (see Takahashi et al., 2013). The CCC was followed by an optional period where students could implement their individualised SDL plans. This second phase was optional and took place outside the classroom because of the LA team’s belief in learner autonomy, and the idea that students need to have some choice in and control over their own learning. It was thought that students who completed the initial CCC would have the tools with which to create an SDL plan anytime in the future.

Table 1. Pilot Course Outline

Content and sequencing

The CCC material used was a mixture of new and existing materials; many of the activities were similar to those that had featured in previous SALC modules, workshops and courses. However, the materials selected were deemed to adequately introduce the learning outcomes (see Takahashi et al., 2013 for details of the learning outcomes). In addition, every attempt was made to ensure that the content adhered to the established principles for content and sequencing for self-directed language learning courses (see appendix A for details). For example, these principles assured the content coverage, opportunities for personalisation, and learnability of the materials.

The sequencing was determined based on experience gained by administering the FSM course.It was carefully monitored during the pilot phase by the LAs reflecting on such matters as the use of the materials and the reactions of the students, using a shared document and also in weekly meetings, in case changes would be necessary for future courses. The CCC (in order) was:

• Using the SALC and the English Language Institute services (Writing Centre, Practice Centre and the Yellow Sofas (a free conversation practice area)
• Analyzing needs, setting and reviewing goals
• Selecting, using and evaluating resources
• Selecting, using and evaluating strategies
• Making, implementing and evaluating a learning plan
• Evaluating linguistic gains

Optional Course Content (OCC) is content which was deemed important, but best introduced organically within the CCC and also made available in a more tailored way to students when relevant, rather than introduced to all module or course-takers at the same time (Lammons, 2013). The two OCC concepts were “Time Management” and “Affective Strategies” and these were introduced to the students through two extra classroom workshops in the second semester at the request of the teacher. In addition, these concepts were introduced to individual students at appropriate times during the piloting phase. 

Assessment

The approach to assessment was guided by the principles for the self-directed language-learning course (see appendix A for details). These ensure that the students are assessed on the core learning outcomes and not language proficiency, that the assessment is consistent between LAs, and that the assessment procedure is practical and realistic for LAs. For the pilot, the students’ work during the first four content sections was not assessed as these were input sessions, and students were not required to provide the evidence of learning at this point; however, students had to complete a minimal amount of work each week in order to receive the extra credits for the course. For those students who had continued with the implementation of the learning plan, additional credits were given based on a simple descriptive rubric relating to the learning outcomes, indicating that students had “approached the standard”, “met the standard”or“exceeded the standard”in each of the learning outcome areas (see Takahashi et al., 2013 (appendix) for details). Further aspects of assessment will be discussed in the next installment of this column, which includes an overview of assessment of SDL, previous assessment tools and methods used, and how these have contributed to the evolution of the assessment procedures of the SALC SDL curriculum.

Piloting the New Curriculum with One Freshman Class

A new advanced-track Freshman English curriculum was being implemented in 2013, so the team took the opportunity to pilot the material in one class of this new course. One Freshman English teacher kindly agreed to participate and made four 90-minute class periods available in May and June 2013. The pilot class consisted of 20 students who all agreed to participate in thestudy. Three LAs (the authors of this paper) used the four class periods to introduce the pilot versions of the CCC activities during the four weeks. Even though this advantageous ratio of LAs to students could not be sustained during a full implementation version (which needs to reach around 900 students), there were benefits to having three LAs involved in the piloting. The main purpose was so that the implementation phase could be observed by three researchers, which proved useful for subsequent evaluation of the pilot. The class was split into three groups of six or seven students with one advisor assuming responsibility for that same group for the remainder of the course. This enabled the LAs to establish close relationships with those students and to become more aware of their particular needs, goals, strengths and weaknesses in order to observe more easily whether the activities were meeting the students’ needs.

Evaluating the Pilot

Focus and tools for evaluation

The pilot phase (Phase 3 in the curriculum modification process shown in Figure 1) was evaluated using various tools that feature in Nation and Macalister’s (2010) “Focus and tools for evaluation of teaching and learning”(p. 129). Five focus areas were chosen from the original list and six tools were identified as the most relevant to the context (shown in Table 2). In the interests of space, three tools (student self-evaluation, student survey, and course evaluation checklist) are discussed below as these have covered all five focus areas.

Table 2. Focus and Tools for the Pilot Phase Evaluation (adapted from Nation and Macalister, 2010)

Analyzing the Results

Student self-evaluation

At the beginning of the pilot phase, students completed a short self-evaluation questionnaire for the purpose of initially evaluating their perceived knowledge of the CCC and subsequently their amount of learning (see appendix B for the questionnaire and the results). This questionnaire comprised two parts: evaluation of existing knowledge of the SALC and LAs, and evaluation of students’previous knowledge and experience of the CCC areas. It also served to raise awareness among the students of the types of skills involved in SDL and reminded students of the opportunities for learning outside the classroom. As a result, it was found that the majority of students (around 70%), although being somewhat familiar with the SALC, were not aware of the CCC areas.

One of the ways in which to gauge whether the content delivery had been successful was to administer the questionnaire again in week 4 at the end of the input sessions to provide a comparison. The results showed that around 80% of students stated that they knew how to utilise the skills, compared to about the same number not knowing previously.

Student survey

Also in week 4, a student survey was conducted in order to evaluate the amount of learning,the quality of learning and teaching, and student satisfaction (see Table 2). However, it must be taken into account that one potential weakness of using a survey, especially after working so closely with the students, was that they may have been inclined to respond with answers that they thought would be favourable to the LAs and the teacher. To allow for this weakness, all questionnaires were completed anonymously. The survey was created based on following three research questions:

• How useful and interesting did students perceive the SALC activities to be?
• What were studentviews on the activities themselves?
• Why did students choose / choose not to implement their learning plans?

1) How useful and interesting did the students perceive the SALC activities to be?

In the first section of the survey, the students were asked to rate whether twelve SALC activities were useful according to the scale:“Very useful”, “Useful”, “Somewhat useful”, “Not useful”. The students perceived most activities to be either “Very useful”or “Useful”. The items that were perceived to be the most useful by the majority of the students were:

• Getting written comments from a learning advisor
• Setting big and small goals
• Evaluating language gain
• Making a learning plan

In addition to the usefulness of the activities, the students indicated their interest in all twelve SALC activities. The students rated most of the activities as “interesting”, but overall, in comparison to “usefulness”, the figure was lower (see appendix C for the results of the student survey section one). As previous research has indicated (Hasegawa & Thornton, forthcoming; Mynard, Takahashi, & Yamaguchi, 2011), students have mixed views on the role of written reflections due to the lack of clear understanding of the purpose and rated “writing reflections each week” as least interesting. However, the results indicated that students value written advising from learning advisors, and it was rated the highest factor for both interest and usefulness in this survey

2) What were students’ views on the activities themselves?

In the second section, the students were asked to rate 17 statements related to the SALC activities bylevel of difficulty and amount of time spent, preference, andeffectiveness and learning gains. Four scales were used: “Strongly agree”, “Agree”, “Disagree”, and “Strongly disagree”.

1. Difficulty level and time spent:

• More than 90% of students felt that the level was adequate and the time spent on the activities in class was sufficient

2. Preference:

• All students enjoyed working with learning advisors
• All except one student wanted to continue working with a learning advisor
• 90% said they want to take an optional SALC module in the future
• 50% would prefer to write reflections electronically

3. Effectiveness and learning gains:

• All students found that studying content through in-class workshops was effective
• 85% believed that they benefitted from discussing ideas with other students
• 95% believed that they were able to apply the concepts to their own learning
• 84% thought that the SALC activities made them think more deeply about how they learn English

3) Why did students choose / choose not to implement their learning plans?

In the last section, 85% said they chose to implement their learning plans and 15% chose not to. Those who decided to continue found their learning plan useful for achieving their language learning goals and believed that the experience would be beneficial. Those who decided not to implement their plans expressed their desire to continue, but explained that their schedule would not allow them to do so.

Pilot evaluation checklist

Evaluation of the pilot design was achieved using the same checklists from Phase 2 when it was used to evaluate the existing First Steps Module (see Lammons, 2013 for details of how the checklists were created). The CCC (orientation and goal-setting, learning resources, learning strategies, learning plan, implementation of learning plan) were evaluated separately using the checklists. This was in order to ensure that all content fulfilled all principles, for instance whether socio-effective, cognitive and metacognitive skills were covered, if students were able to utilise prior knowledge and skills, and whether the amount of content was suitable in terms of workload. The evaluation revealed that the course generally satisfied all principles, but somepoints need to be addressed as identified in Table 3.

Table 3. Strengths and Weaknesses Identified in the Pilot Course Content

 

Conclusions

Based on the evaluation analysis of the five foci, it is fair to say that the pilot course produced positive results and provided both students and LAs with valuable experiences. Although some conditions were privileged in this pilot course (such as having three LAs in one class room, working with a teacher who values SDL skills, and advanced level students), it was an opportunity to see how the curriculum might work in a different format. After reviewing the obtained results, some changes need to be made in order to adequately address students’ needs. Particularly, the fact that there were many students who were not able to continue SDL training with LAs due to time constraints, despite their own willingness, indicates that further collaboration is necessary with freshman English teachers and the university administration, in order to increase opportunities to do so. What is clear, based on the research, is that in-class SDL training gave all of the students the opportunity to learn crucial SDL skills.

Further possibilities for integration of SALC SDL training within freshman English courses still needs discussion not only in the SALC team but also with teachers and course coordinators. Through the pilot experience, the SALC team has developed a better understanding of what a successful program could look like, and will continue searching the ways to promote and develop SDL skills among students across the wider university community.

Notes on the contributors

Satoko Watkins holds an MA in TESOL from Hawai’i Pacific University, USA. Her research interests include learner development and empowerment.

Neil Curry has been teaching in Japan for 8 years and is currently a learning advisor at Kanda University of International Studies. His primary interests are in Foreign Language Anxiety and language advising.

Jo Mynard holds an M.Phil. in Applied Linguistics from Trinity College, Dublin (Ireland) and a Doctorate in TEFL from the University of Exeter (UK). Her research interests are in affect, advising, learner autonomy and CALL.

Acknowledgements

At various stages, the following people have been key members of the project described in this case study: Neil Curry, Yuki Hasegawa, Elizabeth Lammons, Tanya McCarthy, Brian R. Morrison, Jo Mynard, Diego Navarro, Junko Noguchi, Akiyuki Sakai, Keiko Takahashi, Katherine Thornton, Satoko Watkins, and Atsumi Yamaguchi.

References

Hasegawa, Y., & Thornton, K. (Forthcoming). Examining the perspectives of students on a self-directed learning module. In J. Mynard & C. Ludwig (Eds.), Autonomy in language learning: Tools, tasks and environments. Faversham, UK: IATEFL.

Lammons, E. (2013). Principles: Establishing the foundation for a self-access curriculum. Studies in Self-Access Learning Journal, 4(4), 353-366. Retrieved from https://sisaljournal.org/archives/dec13/lammons/

Mynard, J., Takahashi, K., & Yamaguchi, A. (2011). Learning how to learn / Sophomore modules group report. Studies in Linguistics and Language Teaching, 22, 260-262.

Nation, I.S.P., & Macalister, J. (2010). Language curriculum design. London, UK: Routledge.

Takahashi, K., Mynard, J., Noguchi, J., Sakai, A., Thornton, K., & Yamaguchi, A. (2013). Needs analysis: Investigating students’ self-directed learning needs using multiple data sources. Studies in Self-Access Learning Journal, 4(3), 208-218. Retrieved from https://sisaljournal.org/archives/sep13/takahashi_et_al/

Thornton, K. (2013). A framework for curriculum reform: Re-designing a curriculum for self-directed language learning. Studies in Self-Access Learning Journal, 4(2), 142-153. Retrieved from https://sisaljournal.org/archives/june13/thornton/

Appendices

Appendix A(reproduced from Lammons, 2013)

Principles for Format and Presentation, Content and Sequencing and Assessment

Principles – Format and Presentation

Flexibility:

1. Each course/module should have three kinds of content:

(1)  Core Course Content (CCC) which is deemed essential for all students
(2)  Optional Course Content (OCC) which students are provided access to, but are not required to do, and
(3)  Resources and Materials (R&M) which are chosen by the learners and relate to their needs and goals

2. Students should be free to choose their own R&M within the context of the course.
3. Learners should decide how to apply the CCC and OCC to their own learning
4. Syllabuses should cater for different learning styles and preferences
5. Learners should reflect on their learning in both written form and face to face

Compulsory/optional:

6. Any course involving the writing of a learning plan should also include a certain time of implementation (the length of time may vary according to the individual course)
7. Learners should have optional opportunities to continue implementing a plan after a course has been completed.

Input & Experience:

8. Learners should have opportunities to experiment with all of the core content or optional content that they learn about. (No content should be introduced without including such “experimentation” activities.)

Integration with language classes:

9. Some learner training should occur as part of regular language courses (essential concepts to be decided on the basis of our needs analysis and with teachers)

Interaction

10. Learners should be provided with opportunities to interact with other learners and have opportunities to share and learn from each other, in all courses, either face-to-face or online (or both).

Presentation:

11. All input should be comprehensible for the learners
12. Some bilingual support for technical terms should be made available for students who want to use it (glossary etc).
13. The workload for students should be equal each week
14. The workload for each course should be realistic given other requirements on students’ time, and credit awarded (if applicable) 

Principles – Content & Sequencing 

Content:

1. Students should learn the following four different kinds of skills to optimise their learning.

a. Socio-Affective Skills
b. Cognitive Skills
c. Metacognitive Skills
d. Self-Management Skills

• Objectives:

2. Students should have a clear understanding of the aims and objectives of any course at the beginning (for example by sharing Learning Plans, as well as explaining them in course literature/orientation sessions).

Awareness & Control:

3. Students should spend some time exploring and raising their awareness before narrowing down their focus and practicing taking control of their learning.

Learning history:

4. The course should help learners explore and make the most effective use of previous learning experiences and inform LAs about how they have learned languages previously

Implementation:

5. Students should have opportunities to implement what they have learned about in the learner training and reflect on it (combination of input & experience) in a single course

Implementing an outside class plan:

6. Learners should have the opportunity to implement further optional learning plans outside class.

Personalization:

7. The course should provide guidance in and opportunities for personalization of learning (Students should understand how to draw on their preferences and individual differences to personalise the content & have chances to exercise that personalization.)

Teach- & learnability:

8. The teaching of content (input) should take account of when the learners are most ready to learn them (most likely different for different learners so needs to be flexible)

Learning Burden:

9. The amount of content covered (whether as input or implementation of learning plans) should be realistic for a freshman student given their obligations to classes and extracurricular activities

Learning Burden:

10. Some content (e.g. learning strategies, time management strategies – others) should be spread over several weeks rather than delivered in one unit/lesson/chapter/workshop

Spaced retrieval:

11. Students should have increasingly spaced, repeated opportunities to retrieve and give attention to wanted items from learner training in a variety of contexts

Reflection:

12. Students should reflect on each stage of the learning process.
13. Students should receive guidance and training about how to reflect/monitor their work.

Principles-Assessment

Grading/Assessment

Content to be assessed:

1. Students should be assessed on the core learning outcomes introduced in a module/course
2. The core learning outcomes should be identified clearly for each course/module
3. Language proficiency (grammatical accuracy etc) should not be included in assessment
4. Students’ overall assessment should incorporate a participation element, i.e. that they completed a minimum amount of work each week

Evidence:

5. Student assessment should incorporate artifacts (i.e. evidence in the form of written reflections / documentation / portfolio / completed activities)
6. Other assessable evidence will vary depending on the course or module, but could include: written reports, interviews/advising sessions, document analysis of written work or activities, LHL-style learning pack, class/online participation, attendance, and self-assessment

Workload:

7. Any assessment should be practical and realistic for the learning advisor (time-wise)

Consistency:

8. Assessment should be consistent between advisors (using grading rubrics, doing norming sessions, using consistent approaches to penalties for missed work).

Transparency:

9. Clear definitions of terms / metalanguage should be used to assess students, and shared with them
10. Each course should have clear policies about minimum requirements, attendance, late submissions and missing/incomplete work, which should be shared with students and adhered to by all advisors
11. Grading procedures (rubrics, learning plans etc.) should be shared with students at the start of the course or during the orientation, and made clear to them with as much detail as possible.
12. Course outcomes should be clear to students and they should know that the focus is on learning skills rather than linguistic skills

Level of metacognition:

13. For each learning outcome, a level of metacognition should be identified, i.e. “Largely unaware”, “Becoming aware” “Largely aware” “In control” or similar
14. Descriptions of target behaviours and examples will be included on a rubric for assessment purposes
15. A simplified version of the rubric (or a translation) should be made available to students and referred to when giving feedback

Feedback

16. Students should receive ongoing written feedback (written advising) on a regular basis from a learning advisor (at least once every 2 weeks)
17. Students should receive feedback both during and at the end of a course or module
18. Students should receive feedback on their SDLL skills appropriate the stage they are at
19. Non-credit bearing content should still include feedback (even if there is no “grade” or formal assessment)
20. Students should not receive feedback from their learning advisors on their linguistic skills, but will be advised on how they can get this kind of feedback (PC / WC / SALC materials)

Course Evaluation

21. Student surveys should be conducted at the end of each course
22. Student grades should be used to determine whether each course is achieving its objectives for the students who take it
23. LAs and teachers should be invited to give their assessment of course effectiveness on a regular basis
24. A small group of LAs should be in charge of monitoring and modifying each unit of work (course/module) each year

Appendix B

Students Self-evaluation Result: Evaluation of Existing Knowledge of the SALC and the CCC Areas

Number of total answers: Pre-test 19, Post-test 18

Appendix C

Result of Student Survey Section One: How useful and interesting did the students perceived the SALC activities to be?

Number of total answers: 21